System architecture providing redundant components to improve die yields and system reliability

ABSTRACT

A semiconductor device, such as a multiprocessor chip for a computer system, includes a total number of on-board components which is greater than the number of that component required by the system. The chip may be provided with multiple I/O controllers, e.g. more than one controller per I/O interface, and the I/O controllers can act as backups to one another, with failover logic controlling the backup process. In addition, the number of processors formed on the chip may be greater than the number required by the system, allowing multiple levels of redundancy and greater successful manufacturing yields.

BACKGROUND OF THE INVENTION

[0001] This invention relates to a computer system architecture to improve processor die yields and system reliability. In particular, it relates to a system architecture providing redundant processors on a single chip, coupled via I/O (input/output) controllers to I/O pin interfaces, with redundant I/O controllers also being provided relative to the number of I/O interfaces.

[0002] In chip designs in use today, multiple processors may be formed on a single die, along with other circuitry, such as on-board cache, I/O logic and connectors at the edge of the chip, and I/O controllers controlling the flow of data between the I/O interfaces and the processors. For instance, in a typical system in use today as shown in FIG. 1, a number of processors (such as the four processors 20-50 shown) are formed on a die, along with crossbar circuitry 60 coupling the processors to input-output controllers (IOCs) 70-90. The IOCs communicate with I/O interfaces 100-150, with one IOC coupled to each pair of I/O interfaces, as shown.

[0003] The I/O interfaces 100-150 are connected to pins 160 along an edge 170 of chip 10, for coupling to a circuit board, such as the motherboard of a workstation or server.

[0004] In such a system, failure of any one of the processors will make the chip useless, since in general the system in which the chip is used will depend upon having all four (in this example) processors available.

[0005] Likewise, if a given I/O controller on the chip fails, then a path from a processor to an I/O interface is made unavailable, and the chip becomes worthless. Thus, each processor and each IOC forms part of a critical functional unit, which requires replacement of the chip upon failure.

[0006] In semiconductor chip manufacture, a certain nonzero failure rate is inevitable, and regular memory arrays are already routinely repaired. As circuit design becomes more and more complex, and electronic features are shrunk further, the probability of faults in the processed wafers increases, and at the same time the price of failure for a given wafer or die also increases.

[0007] It is therefore becoming more important that architecture features be developed that deal with these factors, and in particular that minimize the high penalties associated with the failure of circuit components on a die, namely the discard of the die and associated computational and economic loss. For example, current processors are vulnerable to die loss if a defect occurs in 70-80% of the die area, since the reparable memory arrays take up the rest of the die.

SUMMARY OF THE INVENTION

[0008] The processor-based architecture of the present invention uses a redundant-processor design, where multiple processors on a die are connected to a crossbar circuit. The crossbar circuit is itself connected to a number of input-output controllers (IOCs), which are in turn connected to I/O interface logic and pins, typically positioned at the edge of the chip to provide communication between the processors and a system motherboard or other circuitry.

[0009] The number of processors on the chip is, in a preferred embodiment, greater than the number of processors required by the system into which the chip is to be installed. Circuitry and software control are provided to determine and keep track of the number of properly functioning processors, both at the time of manufacture and in the operating environment.

[0010] If a processor fails, a saved state for that processor is transferred to a functional processor in a failover process, and the system continues to operate without the need to replace the chip or motherboard.

[0011] Similarly, redundant IOCs are provided between the crossbar circuitry and the I/O interfaces, such that failure of one or more IOCs can be tolerated without interfering with the operation of the system.

[0012] These design features allow both greater yield at time of manufacture and greater reliability of multiprocessor circuitry in operation.

BRIEF DESCRIPTION OF THE DRAWINGS

[0013]FIG. 1 is a block diagram of an architecture according to prior systems.

[0014]FIG. 2 is a block diagram of a new system architecture according to one embodiment of the present invention.

[0015]FIG. 3 is a block diagram of another embodiment of the new system architecture of the present invention.

[0016]FIG. 4 is a block diagram showing details of the control logic for the system shown in FIG. 2.

[0017]FIG. 5 is a block diagram showing details of the control logic for the system shown in FIG. 3.

[0018]FIG. 6 is a block diagram showing yet another embodiment of the new system architecture of the invention.

[0019]FIG. 7 is a block diagram showing details of the control logic for the system shown in FIG. 6.

[0020]FIG. 8 is a block diagram of a multiprocessor system suitable for use in connection with the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] A multiprocessor system chip 200 according to the present invention is shown in FIG. 2. In this embodiment, eight processors 210-245 are formed on the die, and hence on the installed chip, and are connected to a crossbar circuit 270. Multiple I/O controllers (IOCs) 300-335 are likewise connected to the crossbar circuit 270, and these IOCs communicate with I/O interfaces 340-375. The I/O interfaces connect to the system motherboard or another circuit board via pins 400 along edges 410-425 of the chip 200.

[0022] Several features appear in the architecture of FIG. 2 that are different from a prior system such as shown in FIG. 1. First, the chip of FIG. 2 will in general be suitable for use in a system requiring fewer than the total number of processors actually provided on the chip 200. For example, chip 200 may be designed for installation into a system using four or six processors, not the eight total processors actually formed on the chip.

[0023] Thus, the chip 200 includes a control circuit 430, which may incorporate hardware, firmware and/or software control functionality, which governs the actual operation of the processors 210-245 (and is coupled to the processors in a conventional manner). At the time of chip manufacture, in general each chip will be tested to determine whether all of the processors on board are functional. If all eight processors are operational, then six of the eight (in this example) will be selected in a predetermined manner for actual operation in the system for which the chip is designed. For example, the chips may be identified by identification codes, and the lowest-numbered functional chips are selected for default usage in the system.

[0024] For the purposes of this description, “operational” or “functional” means that the component in question has been successfully formed on the chip, so that it is capable of operating essentially in the manner for which it was designed. This is determined by conventional testing methods.

[0025] If one of the eight processors, e.g. processor 230, is found by the manufacturing diagnostic steps as functioning incorrectly, that processor is permanently removed from operation, under control of the circuit 430, and in particular under control of a processor control circuit 440 shown in FIG. 4. This circuit 440 may include, for example, an EEPROM that is programmed with the appropriate information, namely the control software and, in this example, the information that processor 230 is faulty and should not be used. Instead of an EEPROM, some other storage unit (including some other type of PROM, ROM, etc.) could be used.

[0026] In this case, processors 210-225 and 235-240 may be the selected default processors, and processor 245, while in principle operational, is not actually used by the system because six processors are already available.

[0027] However, it may be that during the life of the chip 420, one of the processors 210-225 or 235-240 fails. Processor control circuit 440 includes logic for detecting this, and when the failure occurs circuit 440 removes the faulty processor from operation, preventing further communication with the IOCs or I/O interfaces. The failover logic (or indeed, any of the control circuits referred to herein) may include electronic or semiconductor elements (e.g. processors, logic, ROMs, etc.), firmware or software, or some combination of these as appropriate for the application.

[0028] Circuit 440 at this time brings processor 245 into operation, substituting for the failed processor. Conventional failover techniques may be used, such as maintaining states of the operational processors and transferring a saved state to the newly operational processor upon failure of one of the previously used processors.

[0029] While processors 210-225 and 235-240 are operating, circuit 440 may control processor 245 in various ways. One possibility is to prevent driving the clock of processor 245, thus saving energy and minimizing heat generation while the processor 245 is not needed.

[0030] In another embodiment, processor 245 may be used for real-time diagnostics and/or state-saving computation relative to the operational processors. Alternatively, either or both of these functions may be carried out by the control circuit 440, with the trade-off being that a greater amount of processing power (and circuitry) will in general be needed for such enhanced failover functionality of the circuit 440.

[0031] An advantage of having an otherwise unused processor (such as 245) govern the failover functions is not only that it can simplify and reduce the expense of the circuit 440, but also that, with appropriate distributed software control, any of the unused processors can be used for such operations. Thus, there is less likelihood of failure of the chip, which could result if dedicated control circuitry 440 is used for the important failure detection and recovery operations. Thus, this critical path for high availability can be made more robust by using the redundancy inherent in the architecture of the invention.

[0032] In general, for a system designed to use x processors on a given chip, there will be some number y≧x of operational processors provided on the chip. The die will be designed to form z processors, which will in general be greater than x. Of the z processors, some number y≦z will be diagnosed as properly operational. (In the above example, x=6, y=7 and z=8, and thus x≦y≦z.)

[0033] When a chip is cut from a wafer and tested, the value for y (i.e., the number of operational processors) is determined. This information is provided to the customer, and a variety of chips can be made available, with their respective value relating to the number of backup processors available on board (between zero and z−x). There is a cost associated with providing very many backup processors, but in critical settings a high degree of redundancy may be desired.

[0034] For instance, in the above example, there may be at least twice as many processors on board as actually needed for the system, i.e. z≧2x (or it could be required that y≧2x). An advantage of such a system is that each backup processor may carry out operations in tandem with an associated operational processor, such that a fully redundant highly available system is implemented on a single chip. Additional processors beyond the second full set may be provided as desired, e.g. for further backup functionality (in case of a double failure, i.e. the failure of one of the backup processors) or to provide failover control.

[0035] In the embodiment of FIG. 2, if x=4 then z=2x, and such a fully redundant HA system is presented (for a system requiring a three-processor chip), with appropriate implementation of hardware and software control. For critical settings, the multiplier can be higher; e.g. the designer could choose z=3x or in general z=Mx (or y=Mx), where M can be any number greater than or equal to one. M need not be an integer, as long as the product Mx is an integer or is rounded to an integer (i.e., the number or processors or other elements to be fabricated is a whole number).

[0036] In FIG. 2, it will be noted that the number of IOCs 300-335 is equal to the number of I/O interfaces 340-375. In this case, there are eight IOCs and eight I/O interfaces, with each IOC being connected to two I/O interfaces, and likewise each I/O interface being connected to two IOCs. Thus, a certain amount of redundancy is provided for each I/O channel off the chip, via the pins 400.

[0037] The interconnections between the IOCs and the I/O interfaces, as well as the circuitry between the processors and the IOCS, can be accomplished in a variety of manners, using crossbar designs, conventional multiplexing, or other variable configuration circuitry, e.g. circuit-switched or packet-switched designs or FPGA (field-programmable gate array) elements.

[0038] Using this arrangement, if an IOC—for example, IOC 305—fails, then the control circuit 430, which is coupled to the crossbar 270 and the IOCs, detects this. In particular, control circuit 430 is provided with IOC control logic and programming 450 (see FIG. 4), which monitors the behavior of all of the IOCs 340-375 and controls their connections (e.g. the multiplexing circuitry) to the crossbar 270 and the I/O interfaces 340-375. If IOC 305 fails, then control circuit 450 removes it from the active IOCs in a conventional manner, e.g. by preventing any data from passing to or from the IOC 305.

[0039] In this case, IOC 300 must thereafter handle all input and output through I/O interface 345, and IOC 310 handles all I/O through I/O interface 350. IOCs 310 and 315 still cooperate to ensure I/O through I/O interface 355, i.e. each can still act as a backup to the other for interface 355, although IOC 310 has sole responsibility for interface 350. Similarly, IOCs 300 and 335 continue to act as backups for one another for I/O interface 340, though IOC 300 has sole responsibility for interface 345.

[0040] If IOC 315 were also to fail, then control circuit would provide IOC 300 with connections and responsibility for both interfaces 350 and 355, while IOC 320 would take over all I/O tasks for interface 360.

[0041] In principle, an arbitrarily large amount of redundancy can be provided for the IOCs, by connecting them to a larger number of interfaces. The arrangement in FIG. 2 is easier to manufacture than one with a larger number of interconnections, with a resulting increase in reliability and cost savings due both to its relative simplicity and the higher expectable yield of functional units during fabrication.

[0042]FIG. 3 shows an alternative embodiment of a chip 500 of the invention, again provided with multiple processors 510-560 connected by a crossbar 570 to IOCs 590-650. As in the embodiment of FIG. 2, the IOCs control the transfer of data to and from I/O interfaces 660-710, which communicate off-chip via pins 720 along edge 730 of the chip 500.

[0043] Control of the interconnections, failover, etc. of the chip 500 is governed by control circuit 740, which, as shown in FIG. 5, includes elements such as processor control 750, IOC control 760 and failover control 770. Controls 750-770 operate in a manner similar to that of controls 440-460 shown in FIG. 4, with additional functionality provided as needed for the alternative embodiment of FIG. 3.

[0044] This alternative embodiment includes a switching “fabric” 580, which ideally includes one possible circuit path for each IOC-I/O interface combination. Thus, in the example design shown in FIG. 3, with seven IOCs 590-650 and six I/O interfaces 660-710, there are forty-two paths by which the IOCs can be connected to the I/O interfaces.

[0045] This can be achieved in a variety of manners, including any combination of those mentioned above, such as multiplexing or using FPGAs. If an FPGA circuit is used, then it is controlled by the IOC control 740 in a known manner for FPGAs to make the appropriate IOC-I/O interface connections.

[0046] Using a connection fabric 580 allows each IOC to act as a backup to all the other IOCs, rather than simply backing up in pairs (or other numbers of IOCs), as in the embodiment of FIG. 2. By providing as many IOCs as desired, depending upon the criticality of the application weighed against the expense of additional circuitry, an arbitrarily large amount of redundancy and reliability can be provided for the chip 500.

[0047] The same is true of the processors 510; that is, the crossbar 570 can also be replaced by an interconnect fabric like the IOC fabric 580, with the result that all of the processors back one another up.

[0048] This arrangement provides two levels of redundancy: one at the processor level and one at the IOC level. Depending upon the architecture of the chip, there may be other levels or on-board functions for which it would be helpful to have a connection fabric or variable interconnect design as described above. The architecture of the invention is not confined to IOCs and processors, but is also applicable to such other functions.

[0049] A generalized embodiment of the architecture of the invention appears in FIG. 6, which shows a chip 800 with six processors 810-860 connected to a processor interconnect fabric 870, as described above. The fabric 870 connects via an I/O controller fabric 890 to the I/O interfaces 900, in the manner described with respect to FIG. 3.

[0050] In FIG. 6, the IOC fabric block 890 should be understood to include the I/O controllers; that is, they are not separately represented as in FIGS. 1 and 3. Likewise, the I/O interfaces 900 are not individually shown, but should be understood to include the individual I/O interfaces and pins as described for FIGS. 1 and 3.

[0051]FIG. 6 also shows a separate auxiliary components fabric 910, which includes circuit, processor or other electronic elements, including software or firmware as needed, for any other function that the chip architect may decide to include on-board. For instance, it may be deemed desirable to include redundant circuitry for the on-board cache, which can be implemented in a manner like the redundant processor and IOC implementations described above.

[0052] Interconnect control circuit 920 in FIG. 6 operates in a manner similar to control circuits 430 and 740 shown FIGS. 2-5, to control the connectivity, failover, etc. of the respective components on the chip 800.

[0053] A further extension of the auxiliary components fabric could, for example, implement a COMA (cache-only memory architecture) or NUMA (non-uniform memory architecture) design on-board the processor chip, and the fabric 910 can provide access to all of the processors 810-860 for the caches, with as much redundancy as desired to provide high reliability.

[0054] In general, the features of the present invention can be applied to any on-chip elements, in particular those that use a valuable resource such as a processor or a controller. The redundancy provided for such resources both ensures high reliability for the chip as a whole in operation and the ability to tolerate chip defects at time of manufacture.

[0055] The various control circuits (430, 740, 920, with the individual control modules 440-460, 750-770 and 930-960, respectively) shown in FIGS. 4, 5 and 7 can be configured to control their respectively associated chip elements in a variety of manners. For instance, as indicated above any of them could be programmed or otherwise designed to turn their associated components off or to some extent power them down (i.e., place them in a “wait” or “sleep” state) whenever the components are not in use, in order to save power and minimize heat generation. Generally, a powered-down, non-clocked, or partially powered-down situation may be referred to as a “wait” state for a given component, while a “backup” mode may be considered either a wait state, a fully redundant operational state, or in general any mode in which the backup component is not providing the primary functionality to the system.

[0056] Alternatively, some or all of the components may be powered and clocked in a fully operational backup mode, while others are placed in a wait state, thus both providing the high-availability features (e.g. tandem operation, state preservation, etc.) described above and ensuring a quick failover if that does occur. Finally, all of the redundant components could be kept in active backup mode for the quickest failover possible.

[0057] One variation on this is, in a system with triple redundancy of a component (e.g. the processors), to keep the second tier of redundancy fully operational in tandem, while holding the third tier in a wait state. If individual ones of the second-tier components fail, their respective third-tier backups can be brought into operation, while the remaining third-tier components are kept in the wait state.

[0058] Components in the wait state that are brought into full operation are provided with state information of the system at that time, in a manner known in the art.

[0059] It will be appreciated that, at the time of manufacture, if z components (processors, I/O controllers, or other components) are formed on a chip, but only x components are required by the chip or the system, then if at least x properly functional components are successfully fabricated, that chip can be deemed to have passed quality control. This potentially greatly increases the yield of the fabrication process, since any number of the components beyond x that function are simply a bonus. It may be, as discussed above, that certain settings will require more than x processors, e.g. to provide a highly available design, in which case the principle is the same—to fabricate on the chip a number of components greater than the required number, using any functional excess as desired but thereby providing manufacturing leeway in case some or all of the excess components are not properly functional.

[0060]FIG. 8 shows the architecture for a multiprocessor system 1000 using a multiprocessor chip 1010 (or a chip providing other multiple components, including processors, I/O controllers, and/or other components as described above with respect to FIG. 6). The chip 1010 may thus be essentially like any of the chips 200, 500 or 800 described above.

[0061] The system 1000 includes a motherboard with conventional circuitry 1030 (memory, I/O, etc.), and in addition may include interconnect control logic 1020 into which any or all of the features of the interconnect control 920 (or the controls 430 or 740) may be incorporated. Thus, if the system architect wishes to provide failover functionality for the chip's components at the system level, this is possible, and either the on-chip control logic, or the control logic at the system level, or both, may be used.

[0062] Thus any one or more of the component verification, state maintenance, failover protocols, etc. may all be provided at the system level, if desired, so long as the system is provided with sufficient information to identify, test for functionality and control the operation of the individual components on board the chip 1010. To this, end, each of the desired components may be provided with a unique ID for use in communicating with the system-level logic 1020 for such verification and operational control.

[0063] Other embodiments incorporating the inventive features are possible. 

What is claimed is:
 1. A semiconductor device configured for use in cooperation with a processor-based system, including: a first number of components formed on the semiconductor device; and a plurality of communication channels configured to transfer data between respective ones of the components and other elements of the system; wherein the semiconductor device is configured for operation in cooperation with the system using a second number of the components, the second number being less than the first number.
 2. The device of claim 1, further including a controller configured to control operations of the components.
 3. The device of claim 2, wherein the operations include an operation of placing at least one component in a wait state.
 4. The device of claim 1, wherein: the components include microprocessors; and the communication channels include I/O paths connected to at least one I/O interface for the semiconductor device.
 5. The device of claim 1, wherein: the components include I/O controllers; and the communication channels include I/O paths connected to at least one I/O interface for the semiconductor device.
 6. The device of claim 1, wherein at least one of the components beyond the first number of components is provided as a redundant component to the first number of components.
 7. A semiconductor device configured for use in cooperation with a processor-based system, including: a number z of processors formed on the semiconductor device; and a plurality of I/O controllers connected to I/O interfaces for the semiconductor device and configured to control data transfer between the processors and the interfaces; wherein the device is configured to provide a number x of operational processors for use by the processor-based system, and where the number z is greater than the number x, thereby providing a number z−x of redundant processors.
 8. The device of claim 7, further including failover logic configured to determine when one of the x processors has failed, and upon such determination to provide one of the z−x redundant processors for operation with the processor-based system.
 9. The device of claim 7, further including logic configured to place at least one of the z−x redundant processors in a wait state.
 10. A method of manufacturing a semiconductor device for use in cooperation with a processor-based system, including the steps of: forming a number z of components on the device; forming control logic on the device configured to provide a number x<z of operational components to operate in cooperation with the system.
 11. The method of claim 10, further including the steps of: testing the z processors to determine a number y of functional components; and determining whether the number y is at least as great as the number x, to determine whether the semiconductor device is usable for operation with the system.
 12. The method of claim 10, wherein the components are microprocessors configured for use in a multiprocessor system.
 13. The method of claim 10, wherein the components are I/O controllers configured to control I/O for the semiconductor device.
 14. The method of claim 10, further including the step of configuring the control logic to place at least one of the components beyond the first x operational components in a wait state.
 15. A processor-based system configured to operate using at least a number x of components provided on a semiconductor device including a number z>x of the components, the system including: control logic to determine whether at least x of the components are operational, and if so, to hold any excess number of components beyond the x operational components in a backup mode.
 16. The system of claim 15, wherein: the processor-based system includes the semiconductor devide, and the control logic is formed at least in part on the semiconductor device.
 17. The system of claim 16, wherein the control logic is at least in part provided external to the semiconductor device.
 18. The system of claim 15, wherein the components comprise a number y of operational processors formed on the semiconductor device, where y≧x, thereby providing a number (y−x)≧0 of backup processors for the system.
 19. The system of claim 15, wherein the components comprise a number y of operational I/O controllers formed on the semiconductor device, where y≧x, thereby providing a number (y−x)≧0 of backup I/O controllers for the system.
 20. The system of claim 15, wherein the control logic is configured to place the excess number of components in a wait state. 